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For  a ti«e  during  the  60's,  I frequently  heard  huiorous 
antecdotes  from  people  around  Cambridge  about  a computer  program, 
ELIZA,  which  parodied  the  behavior  of  a passive  psychotherapist. 
Although  I never  had  the  privilege  of  interacting  with  ELIZA,  I 
thought  of  the  program  as  a rather  clever  and  elaborate  but  good 
\natured  "joke  about  psychotherapy. 

^4  Having  read  a book  (Computer  Power  and  Human  Reason  by  Joseph 
Heizenbaum,  W.  H.  Freeman  and  Company,  San  Fraucisco,  1976),  by 
the  program’s  creator,  it  is  no  longer  clear  to  me  who  the  joke 
was  on.  Apparently,  some  people  took  the  program  seriously^,  A 
number  of  users  of  the  program  attempted  to  obtain  psychotherapy 
froa  it,  and  at  least  one  psychoanalyst  heralded  it  as  an 
existence  proof  tor  the  feasibility  of  automatic  psychotherapy. 
Weizenbaum  was  very  disturbed  by  these  reactions,  disturbed 
enough  to  write  this  book. 

^The  book’s  principal  avowed  purpose  is  to  show  that  there  are 
certain  activities  that  computers  (because  they  are  computers) 
ought  not  to  be  made  to  engage  in  and  derivatively  that  there  are 
certain  kinds  of  research  that  ought  not  to  be  done. 

In  this  endeavor,  the  author  has,  I think,  substantially 
failed.  ore  specifically,  he  has  not  furnished  any  kind  of  a 
structure  which  one  could  use  to  decide  what  kinds  of  activities 
ought  not  to  be  engaged  in  or  researched  about.  Indeed,  I would 
be  totally  unable  to  predict  with  any  confidence  whatever  what 
activities  Weizenbaum  would  forbid  and  which  he  would  not  forbid. 
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The  only  two  specific  activities  (except  psychoanalysis)  he 
would  forbid  to  computers  do  not  appear  until  the  last  chapter  of 
the  book;  they  are  given  as  examples  and  are  both  research 
activities;  his  reasons  for  forbidding  the*  seen  to  be  totally 
unrelated  to  the  argunent  of  his  book. 

The  first  area  of  research  he  would  forbid  is  work  aimed  at 
the  development  of  speech  recognition  systems  (systems  which 
could  convert  oral  speech  to  e.g.  written  form).  His  "argument" 
that  such  research  is  immoral,  as  nearly  as  I can  follow  it,  is 


(1)  that  speech  recognition  capability  implies  natural 
language  understanding, 

(2)  that  any  realization  of  the  capability  would  involve  such 
expensive  computation  that  only  government  could  afford  it, 

(3)  that  government's  use  of  this  technology  might  well  be  to 
monitor  all  telephone  conversations  reporting  only  the 
"interesting"  ones  to  the  government  masters. 

This  argument  is  so  weak,  and  Weizenbaum's  intellect  is  so 
strong,  that  one  can  only  conclude  that  Weizenbaum  has  an 
extremely  strong  emotional  reaction  against  this  work  for  reasous 
which  are  not  clear  to  this  reviewer.  It  is  obvious  that  the 
real  culprit  in  Weizenbaum ' s scenario  of  doom  is  not  speech 
recognition  per  se  (in  fact,  much  of  the  speech  recognition  work 
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is  based  on  extremely  restricted  vocabulary  and  syntax  and 
requires  no  more  natural  language  understanding  capability  than 
does  a FOHTRAN  compiler).  No,  the  real  culprit  is  wcrk  toward 
language  understanding,  but  Weizenbaum  makes  it  clear  that  this 
work  is  not  immoral.  In  fact,  he  likes  it.  Furthermore,  it  is 
obvious  that  once  one  had  natural  language  understanding,  speech 
recognition  would  be  a relatively  trivial  step. 

It  is  also  interesting  to  me  that  a society  possessing  the 
kind  of  technology  that  would  make  it  economically  feasible  for 
computers  to  listen  to,  to  understand,  to  analyze  and  make 
judgments  about  millions  of  telephone  conversation  daily,  is  so 
beyond  the  imagination  of  any  living  human  being  that  any  moral 
judgments  about  how  what  we  might  do  now  would  affect  such  a 
society  are  silly. 

Weizenbaum  dismisses  the  euormous  good  such  a technology 
would  bestow,  (for  example,  dealing  with  the  medical  record 
problem)  without  meaningful  comment. 

Another  kind  of  developmental  research  which  Weizenbaum  finds 
immoral  is  the  coupling  of  an  animal's  visual  system  and  brain  to 
computers.  He  finds  this  work  so  immoral  that  all  he  can  say  is 
that  it  is  disgusting  and  obscene,  and  an  attack  on  life  itself. 
That  what  might  be  learned  in  such  experimentation  might 
ultimately  be  used  to  alleviate  human  pain  and  suffering  to 
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improve  the  quality  and  quantity  of  life,  he  oust  realize,  but 
does  not  mention.  Perhaps  what  he  fears  most  is  that  nan  was 
never  aeant  to  possess  sole  knowledge. 

Thus  the  book  fails  in  its  avcwed  purpose.  But  if  instead 
one  views  it  as  an  exposition,  not  about  computers  and  human 
reason  but  about  Weizenbaum;  that  is,  as  a description  of  the 
thougnts  and  feelinqs  about  cooiputers  of  one  who  has  thought  and 
felt  long  and  hard,  deeply  and  well  about  computers,  then  it  is  a 
superb  book.  It  is  full  of  insight,  charm,  wit  (the  natenal  on 
compulsive  programming  had  me  crying  simultaneously  with  laughter 
and  sorrow),  passion,  learning  and  wisdom. 

Weizenbaum  uses  one  tactic  which  I believe  is  grossly  unfair 
and  misleading.  He  takes  computerniks  at  their  word.  That  is, 
he  confuses  what  they  say  they  have  done,  will  do  and  can  do  with 
what  they  in  fact  have  done,  will  do,  and  can  do,  and  actually 
draws  conclusions  from  the  explication  of  our  group  fantasies. 

But  Weizenbaum  well  knows  that  computerniks  as  a group,  most 
flagrantly  those  in  artificial  intelligence,  speak  and  write  in  a 
special  code  that  it  is  grossly  unfair  to  take  literally.  In  the 
real  world  where  computers  and  programs  must  be  sold,  grants  must 
be  awarded,  papers  must  be  refereed,  and  doctoral  theses  must  be 
approved,  it  is  hard  to  see  how  to  control  this  “inflation  01 
language"  in  which  claims  are  moving  ever  faster  than 
achievements.  The  individual  vendor,  grant  applicant,  author  and 
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candidate  who  started  to  tell  the  truth  would  find  his  words 


discounted  and  his  offerings  to  be  thought  trivial.  Although  it 
is  reasonable  for  Weizenbaum  to  laaent  such  lack  of  candor,  it  i 
not  reasonable  for  hia  to  draw  consequences  froa  literal 
statements. 

One  kind  of  dangerous  program  that  Weizenbaua  points  to  is 
the  very  complex  program  which  is  in  principle  undebuqgable.  By 
very  complex,  I mean  totally  beyond  the  kin  of  any  one  human 
being.  Programs  which  are  very  complex  in  this  sense  have  been 
coiaon  for  quite  soae  tiae.  They  become  dangerous  if  there  are 
no  external,  concrete  easily  agreed  upon  standards  by  which  thei 
performance  aight  be  evaluated,  but  are  used  to  perfora  critical 
tasks.  Graoted  that  many  programs  are  too  complex  to  be  ever 
totally  bug  free,  aost  such  prograas  such  as  compilers  and 
payroll  processors,  can  be  judged  by  reseasonably  objective 
criteria.  But  what  about  programs  designed  to  do  automatic 
diagnosis,  triage,  referral  or  administer  psychotherapy  which 
ace  generated  by  heuristic  programming,  by  many  people  over  long 
periods  of  tiae  and  are  too  coaplex  for  any  human  being  to  have 
any  inkling  of  their  operation  AND  which  can  be  judged  only  by 
their  past  performance  in  comparison  to  that  of  human  beings 
which  is  itself  coaplex  and  unreproducible.  Weizenbaum  fears 
with  iegitiaacy  that  "bugs"  in  such  a program  could  be  very 
dangerous.  Perhaps  the  same  might  be  said  of  human  beings  who 
are  given  such  awesome  responsibility. 
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whatever  else  can  be  said  of  this  book,  I guarantee  that 
reading  it  will  be  an  experience  not  soon  to  be  forgotten,  that 
reaction  to  it  will  be  intense. 


